Remote microphone devices for auditory prostheses

ABSTRACT

Presented herein are techniques for adapting settings/operations of a remote microphone device associated with an auditory prosthesis based on a desired/preferred listening direction of a recipient of the auditory prosthesis. More specifically, an auditory prosthesis worn by a recipient and a remote microphone device, which are configured to wirelessly communicate with one another, are both positioned in the same spatial area. At least one of a recipient-specified (e.g., recipient-preferred) region of interest within the spatial area, or a recipient-specified listening direction, is determined. Based on a determined relative positioning (e.g., location and orientation) of the remote microphone device and the auditory prosthesis, operation of the remote microphone device is dynamically adapted so that the remote microphone device can focus on (e.g., have increased sensitivity to) sounds originating from the recipient-specified region of interest/listening direction.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional PatentApplication No. 62/681,194, filed Jun. 6, 2018, the contents of which ishereby incorporated by reference herein.

BACKGROUND Field of the Invention

The present invention relates generally to remote microphone devicesoperable with auditory prostheses.

Related Art

Hearing loss is a type of sensory impairment that is generally of twotypes, namely conductive and/or sensorineural. Conductive hearing lossoccurs when the normal mechanical pathways of the outer and/or middleear are impeded, for example, by damage to the ossicular chain or earcanal. Sensorineural hearing loss occurs when there is damage to theinner ear, or to the nerve pathways from the inner ear to the brain.

Individuals who suffer from conductive hearing loss typically have someform of residual hearing because the hair cells in the cochlea areundamaged. As such, individuals suffering from conductive hearing losstypically receive an auditory prosthesis that generates motion of thecochlea fluid. Such auditory prostheses include, for example, acoustichearing aids, bone conduction devices, and direct acoustic stimulators.

In many people who are profoundly deaf, however, the reason for theirdeafness is sensorineural hearing loss. Those suffering from some formsof sensorineural hearing loss are unable to derive suitable benefit fromauditory prostheses that generate mechanical motion of the cochleafluid. Such individuals can benefit from implantable auditory prosthesesthat stimulate nerve cells of the recipient's auditory system in otherways (e.g., electrical, optical and the like). Cochlear implants areoften proposed when the sensorineural hearing loss is due to the absenceor destruction of the cochlea hair cells, which transduce acousticsignals into nerve impulses. An auditory brainstem stimulator is anothertype of stimulating auditory prosthesis that may be proposed when arecipient experiences sensorineural hearing loss due to damage to theauditory nerve.

SUMMARY

In one aspect, a method is provided. The method comprises: determining arelative location and orientation of a remote microphone device to anauditory prosthesis, wherein the auditory prosthesis and the remotemicrophone device are each located within a same spatial area andwherein the remote microphone device comprises a plurality ofmicrophones; determining a recipient-specified region of interest withinthe spatial area; and focusing, by the remote microphone device, theplurality of microphones of the remote microphone device on therecipient-specified region of interest within the spatial area.

In another aspect, a method is provided. The method comprises:synchronizing location and orientation information of a remotemicrophone device with location and orientation information of anauditory prosthesis worn by a recipient, wherein the auditory prosthesisand the remote microphone device are each located within a same spatialarea; determining substantially a real-time recipient-specifiedlistening direction for the remote microphone device; and wirelesslysending directional data to the remote microphone device indicting thesubstantially real-time recipient-specified listening direction.

In another aspect a remote microphone device is provided. The remotemicrophone device comprises: a wireless transceiver configured fordirect or indirect communication with an auditory prosthesis; amicrophone array configured to capture sound signals; and at least oneprocessor configured to process the captured sound signals for wirelesstransmission by the wireless transceiver to the wireless transceiver ofthe auditory prosthesis, wherein the at least one processor isconfigured to dynamically adjust a directionality of the microphonearray to one of a recipient-specified area of interest within a spatialarea of the remote microphone device or a recipient-specified listeningdirection for the remote microphone device.

In another aspect an auditory prosthesis system is provided. Theauditory prosthesis system comprises: an auditory prosthesis configuredto be worn by a recipient and comprising a first microphone array; aremote microphone device comprising a second microphone array andlocated in a same spatial area as the auditory prosthesis; and one ormore processors configured to: synchronize location and orientationinformation of the remote microphone device with location andorientation information of the auditory prosthesis; determine, based onthe synchronized location and orientation information, arecipient-specified region of interest within the spatial area; andfocus one or both of the first microphone array or the second microphonearray on the recipient-specified region of interest within the spatialarea.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention are described herein in conjunctionwith the accompanying drawings, in which:

FIG. 1A is a schematic diagram illustrating a cochlear implant systemcomprising a cochlear implant and remote microphone device, inaccordance with certain embodiments presented herein;

FIG. 1B is a block diagram of the cochlear implant of FIG. 1A;

FIG. 1C is a block diagram of the remote microphone device of FIG. 1A;

FIG. 2 is high-level flowchart of a method, in accordance with certainembodiments presented herein;

FIG. 3 is a schematic diagram illustrating use of a reference point witha spatial area, in accordance with certain embodiments presented herein;

FIG. 4 is a schematic diagram illustrating a simplified user interfacefor use in certain embodiments presented herein;

FIG. 5 is a flowchart of an example method, in accordance with certainembodiments presented herein;

FIG. 6A is a schematic diagram illustrating a remote microphone device,in accordance with certain embodiments presented herein; And

FIG. 6B is a schematic diagram illustration operation of the remotemicrophone device of FIG. 6A, in accordance with certain embodimentspresented herein.

DETAILED DESCRIPTION

Presented herein are techniques for adapting settings/operations of aremote microphone device associated with an auditory prosthesis based ona desired/preferred listening direction of a recipient of the auditoryprosthesis. More specifically, an auditory prosthesis worn by (e.g.,positioned on, or implanted in) a recipient and a remote microphonedevice, which are configured to wirelessly communicate with one another,are both positioned in the same spatial area. At least one of asubstantially real-time recipient-specified (e.g., recipient-preferred)region of interest within the spatial area, or a substantially real-timerecipient-specified listening direction, is determined. Based on adetermined relative positioning (e.g., location and orientation) of theremote microphone device and the auditory prosthesis, operation of theremote microphone device is dynamically adapted so that the remotemicrophone device can focus on (e.g., have increased sensitivity to)sounds originating from the recipient-specified region ofinterest/listening direction.

Merely for ease of description, the techniques presented herein areprimarily described herein with reference to one illustrativeauditory/hearing prosthesis, namely a cochlear implant. However, it isto be appreciated that the techniques presented herein may also be usedwith a variety of other non-implantable, partially-implantable, orfully-implantable auditory prostheses, including acoustic hearing aids,bone conduction devices, middle ear auditory prostheses (middle earimplants), direct acoustic stimulators, auditory brain stimulators,devices for a person with normal hearing, etc.

Shown in FIGS. 1A, 1B, and 1C is an exemplary cochlear implant system101 configured to execute the techniques presented herein. Moreparticularly, FIG. 1A is a schematic diagram illustrating that theexemplary cochlear implant system 101 comprises a cochlear implant 100and a remote microphone device 103. FIG. 1B is a block diagramillustrating one example arrangement of the cochlear implant 100, whileFIG. 1C is a block diagram illustrating one example arrangement of theremote microphone device 103. For ease of illustration, FIGS. 1A and 1Bwill be described together, followed by a description of FIG. 1C. Incertain embodiments, the cochlear implant system 101 may also comprise amobile computing device which, for ease of illustration has been omittedfrom FIGS. 1A-1C.

Referring first to FIGS. 1A and 1B, the cochlear implant 100 comprisesan external component 102 and an internal/implantable component 104. Theexternal component 102 is configured to be worn by the recipient (e.g.,directly or indirectly attached to the body of the recipient) andtypically comprises an external coil 106 and, generally, a magnet (notshown in FIG. 1) fixed relative to the external coil 106. The externalcomponent 102 also comprises one or more input elements/devices 113 forreceiving input signals at a sound processing unit 112. In this example,the one or more input devices 113 include sound input devices 108 (e.g.,microphones positioned by auricle 110 of the recipient, telecoils, etc.)configured to capture/receive input signals, one or more auxiliary inputdevices 109 (e.g., audio ports, such as a Direct Audio Input (DAI), dataports, such as a Universal Serial Bus (USB) port, cable port, etc.), anda wireless transmitter/receiver (transceiver) 111, each located in, on,or near the sound processing unit 112.

The wireless transceiver 111 may have a number of differentarrangements. In one example, the wireless transceiver 111 includes anintegrated antenna 117 and may be configured to operate in accordancewith the Bluetooth® or other short-range wireless technology standardthat enables the sound processing unit 112 to wirelessly communicatewith another device (i.e., receive and transmit data to/from anotherdevice via a wireless connection using, for example, 2.4 Gigahertz (GHz)Ultra high frequency (UHF) radio waves, 5 GHz Super high frequency (SHF)radio waves, etc.). Bluetooth® is a trademark of Bluetooth SpecialInterest Group (SIG), Inc. It is to be appreciated that reference to theBluetooth® standard is merely illustrative and that the wirelesstransceiver 111 may make use of any other wireless standard now known orlater developed.

Sound processing unit 112 may also comprises one or moreorientation/directional sensors 135 (e.g., one or more of anaccelerometer, a gyroscope, a magnetometer, a compass, etc.). Inaddition, the sound processing unit 112 includes, for example, at leastone power source (e.g., battery) 107, a radio-frequency (RF) transceiver121, and a processing module 125 that includes a sound processing engine123 and a remote microphone focusing engine 127. The processing module125, and thus the sound processing engine 123 and the remote microphonefocusing engine 127, may be formed by any of, or a combination of, oneor more processors (e.g., one or more Digital Signal Processors (DSPs),one or more uC cores, etc.), firmware, software, etc. arranged toperform operations described herein. That is, the processing module 125may be implemented as firmware elements, partially or fully implementedwith digital logic gates in one or more application-specific integratedcircuits (ASICs), partially or fully in software, etc.

In the examples of FIGS. 1A and 1B, the external component 102 comprisesa behind-the-ear (BTE) sound processing unit 112 configured to beattached to, and worn adjacent to, the recipient's ear and a separatecoil 106. However, it is to be appreciated that embodiments of thepresent invention may be implemented with systems that include otherarrangements, such as systems comprising an off-the-ear (OTE) soundprocessing unit (i.e., a component having a generally cylindrical shapeand which is configured to be magnetically coupled to the recipient'shead and which includes an integrated coil), a mini or micro-BTE unit,an in-the-canal unit that is configured to be located in the recipient'sear canal, a body-worn sound processing unit, etc.

Returning to the example embodiment of FIGS. 1A and 1B, the implantablecomponent 104 comprises an implant body (main module) 114, a lead region116, and an intra-cochlear stimulating assembly 118, all configured tobe implanted under the skin/tissue (tissue) 105 of the recipient. Theimplant body 114 generally comprises a hermetically-sealed housing 115in which RF interface circuitry 124 and a stimulator unit 120 aredisposed. The implant body 114 also includes an internal/implantablecoil 122 that is generally external to the housing 115, but which isconnected to the RF interface circuitry 124 via a hermetic feedthrough(not shown in FIG. 1B).

Stimulating assembly 118 is configured to be at least partiallyimplanted in the recipient's cochlea 137. Stimulating assembly 118includes a plurality of longitudinally spaced intra-cochlear electricalstimulating contacts (electrodes) 126 that collectively form a contactor electrode array 128 for delivery of electrical stimulation (current)to the recipient's cochlea. Stimulating assembly 118 extends through anopening in the recipient's cochlea (e.g., cochleostomy, the roundwindow, etc.) and has a proximal end connected to stimulator unit 120via lead region 116 and a hermetic feedthrough (not shown in FIG. 1B).Lead region 116 includes a plurality of conductors (wires) thatelectrically couple the electrodes 126 to the stimulator unit 120.

As noted, the cochlear implant 100 includes the external coil 106 andthe implantable coil 122. The coils 106 and 122 are typically wireantenna coils each comprised of multiple turns of electrically insulatedsingle-strand or multi-strand platinum or gold wire. Generally, a magnetis fixed relative to each of the external coil 106 and the implantablecoil 122. The magnets fixed relative to the external coil 106 and theimplantable coil 122 facilitate the operational alignment of theexternal coil with the implantable coil. This operational alignment ofthe coils 106 and 122 enables the external component 102 to transmitdata, as well as possibly power, to the implantable component 104 via aclosely-coupled wireless link formed between the external coil 106 withthe implantable coil 122. In certain examples, the closely-coupledwireless link is a radio frequency (RF) link. However, various othertypes of energy transfer, such as infrared (IR), electromagnetic,capacitive and inductive transfer, may be used to transfer the powerand/or data from an external component to an implantable component and,as such, FIG. 1B illustrates only one example arrangement.

The processing module 125 of sound processing unit 112 is configured toperform a number of operations. In particular, the processing module 125is configured to convert sound/audio signals into stimulation controlsignals 136 for use in stimulating a first ear of a recipient (i.e., thesound processing engine 123 is configured to perform sound processing oninput audio signals received at the sound processing unit 112). Thesound signals that are processed and converted into stimulation controlsignals may be sound signals received via the sound input devices 108,signals received via the auxiliary input devices 109, and/or signalsreceived via the wireless transceiver 111.

In the embodiment of FIG. 1B, the stimulation control signals 136 areprovided to the RF transceiver 121, which transcutaneously transfers thestimulation control signals 136 (e.g., in an encoded manner) to theimplantable component 104 via external coil 106 and implantable coil122. That is, the stimulation control signals 136 are received at the RFinterface circuitry 124 via implantable coil 122 and provided to thestimulator unit 120. The stimulator unit 120 is configured to utilizethe stimulation control signals 136 to generate electrical stimulationsignals (e.g., current signals) for delivery to the recipient's cochleavia one or more stimulating contacts 126. In this way, cochlear implant100 electrically stimulates the recipient's auditory nerve cells,bypassing absent or defective hair cells that normally transduceacoustic vibrations into neural activity, in a manner that causes therecipient to perceive one or more components of the input audio signals.

As noted, FIGS. 1A, and 1B illustrate one example arrangement for thecochlear implant 100. However, it is to be appreciated that embodimentsof the present invention may be implemented in cochlear implants orhearing prostheses having alternative arrangements. For example, it isto be appreciated that the use of an external component is merelyillustrative and that the techniques presented herein may be used inarrangements having an implanted sound processor (e.g., totallyimplantable cochlear implants, etc.). It is also to be appreciated thatthe individual components referenced herein, e.g., sound input element108 and the sound processor in sound processing unit 112, may bedistributed across more than one prosthesis, e.g., two cochlearimplants, and indeed across more than one type of device, e.g., cochlearimplant 100 and a consumer electronic device or a remote control of thecochlear implant 100.

Increasingly popular with recipients/users of auditory prostheses (e.g.,hearing aids, cochlear implants, etc.) are remote/wireless microphonedevices/accessories. Remote microphone devices are stand-alone portableelectronic components (e.g., physically separate from an auditoryprosthesis) that include one or more microphones, and possibly othersound input devices, configured to capture/receive sound signals. Thecaptured sound signals are then digitally encoded and wirelesslytransmitted to the auditory prostheses.

There are many applications for these remote microphone devices. Oneapplication is where an auditory prosthesis recipient leaves the remotemicrophone device on the lectern during a lecture. Due to the muchcloser proximity of the remote microphone device to the lecturer, themicrophone(s) on the remote microphone device then capture the audiofrom the lecturer far better than the auditory prosthesis itself wouldwhen worn by the recipient sitting in the audience. The audio that theremote microphone device picks up from the lecturer at the lectern isstreamed wirelessly to the auditory prosthesis.

Remote microphone devices work well in the above application and othersin which the remote microphone device is held in a fixed position, suchas when the device is worn on a speaker (e.g., the speaker's shirtpocket or collar), and the fixed directionality of the microphones istoward the sound source (e.g., speaker's mouth/head). However, there areseveral applications of remote microphone devices where the assumptionthat the fixed directionality of the microphones is toward the soundsource may not necessarily be satisfied. For example, consider asituation in which the remote microphone device is placed on a tableduring a conference or meeting, in order to ensure that the remotemicrophone device is in reasonable proximity to all potential speakers,and any telephone systems, involved in the conference. However, in suchsituations, conventional remote microphone device may capture soundsfrom all areas around the table the same, only capture sounds from fixedareas, or utilize a signal-to-noise ratio-based (SNR-based) approach tocapture only the loudest sounds. In each of these examples, the remotemicrophone device fails to take into account what specific sounds, orsound sources, the recipient actually prefers/desires to hear (e.g.,conventional devices do not take the recipient's specific preferencesinto account, may have a general disadvantage from not being placed inthe best possible location, and/or are reliant on SNR to capturesounds).

For example, again consider the situation where the remote microphonedevice is placed on a table during a conference call. The microphones ofthe remote microphone device may be directed towards the telephone(either via physical positioning of the remote microphone device or dueto the fact that the telephone is the loudest sound source in the room).However, the recipient may need to engage in a side conversation withanother person sitting at the table. In this situation, the recipientwould prefer to focus on sounds from the other person engaged in theside conversation, rather than sounds emanating from the telephone.However, this is not possible with conventional remote microphonedevices that have fixed directionality or which use an SNR-based tocapture the loudest sounds. Presented herein are techniques that addressthese problems with conventional devices by enabling a remote microphonedevice to dynamically focus on (e.g., have increased sensitivity to)sounds originating from a real-time (i.e., dynamically determined withor without some hysteresis, including attack/release times)recipient-specified region of interest or real-time recipient-specifiedlistening direction. The techniques presented herein may, for example,improve the usage of remote microphone devices in situations such asmeetings, conference rooms, classrooms, and lectures since the remotemicrophone devices will be able to optimally steer the microphone beamin the recipient's desired direction of listening, and do sodynamically.

More particularly, returning to the example of FIGS. 1A-1C, cochlearimplant system 101 includes a remote microphone device 103 that isconfigured to address the problems with conventional remote microphonedevices. As shown in FIG. 1C, remote microphone device 103 comprises aplurality of microphones 140 forming a microphone array 141. The remotemicrophone device 103 also comprises at least one adaptive processor142, a battery 144, a wireless transceiver 145, a user interface 146,and one or more orientation/directional sensors 148 (e.g., one or moreof an accelerometer, a gyroscope, a magnetometer, a compass, etc.).

In operation, sounds are captured/received by the microphones 140 andconverted to electrical signals 147. Although not shown in FIG. 1C, theremote microphone device 103 could also comprise other sound inputdevices in addition to the microphones 140, such as a telecoil, audioinput port, etc. The electrical signals 147 output by the microphones140 (or other sound input devices) are provided to the adaptiveprocessor 142. The adaptive processor 142 may be configured to performone or more processing operations on the electrical sounds 147, such as,for example, filtering, equalization etc.

As described further below, the adaptive processor 142 is alsoconfigured to dynamically adapt operations performed on the electricalsignals 147 based on a determined recipient-specified region of interestor recipient-specified listening direction. For example, the adaptiveprocessor 142 is configured to change the one or more processingoperations performed on the electrical signals 147 so as to focus themicrophone array 141 (i.e., change the directionality/sensitivity of themicrophone array 141) to a recipient-specified region ofinterest/listening direction. That is, the adaptive processor 142 isconfigured to dynamically process the sound signals (e.g., theelectrical signals 147) in a manner such that sounds present in and/oremanating from a limited/constrained region of the physical environmentsurrounding the remote microphone device 103 are emphasized relative tosounds present in and/or emanating from other areas of the of thephysical environment surrounding the remote microphone device 103.

As described further below, the constrained region from which sounds arecaptured is “recipient-specified,” meaning that the region is selectedbased on one or more recipient provided indicia. Thedirectionality/sensitivity of the microphone array 141 (i.e., the focusof the remote microphone device 103) to the recipient-specified regionof interest/listening directional may be achieved through beamformingoperations (e.g., the adaptive processor 142 may comprise one or moremicroprocessors capable of beam-forming to enhance the received audio indirection desired by the recipient).

As a result of the above operations, the adaptive processor 142generates processed electrical signals 149 which emphasize the soundscaptured from the recipient-specified region of interest/listeningdirection. The processed electrical signals 149 are provided to thewireless transceiver 145 which is then configured to wirelessly send theprocessed electrical signals 149 to the sound processing unit 112 (i.e.,via the wireless transceiver 111).

Similar to the wireless transceiver 111, the wireless transceiver 145may also have a number of different arrangements. In one example, thewireless transceiver 145 includes an integrated antenna 139 and may beconfigured to operate in accordance with the Bluetooth® or othershort-range wireless technology standard that enables the remotemicrophone device 103 to wirelessly communicate with sound processingunit 112 to another device (i.e., receive and/or transmit data to/fromanother device via a wireless connection using, for example, 2.4Gigahertz (GHz) Ultra high frequency (UHF) radio waves, 5 GHz Super highfrequency (SHF) radio waves, etc.). Bluetooth® is a trademark ofBluetooth Special Interest Group (SIG), Inc. It is to be appreciatedthat reference to the Bluetooth® standard is merely illustrative andthat the wireless transceiver 145 may make use of any other wirelessstandard now known or later developed.

The adaptive processor 142 may be formed by any of, or a combination of,one or more Digital Signal Processors (DSPs), one or more uC cores,etc., firmware, software, etc. arranged to perform operations describedherein. That is, the adaptive processor 142 may be implemented asfirmware elements, partially or fully implemented with digital logicgates in one or more ASICs, partially or fully in software, etc. Theuser interface 146 may take many different forms and may include, forexample, one or more buttons, touchwheel, touchscreen, etc. In oneexample, the user interface 146 includes one or more light emittingdiodes (LEDs) or other devices that provide a visual indication of thedirection in which the microphone array 141 is focused.

It is to be appreciated that FIG. 1C illustrates one example arrangementfor a remote microphone device. However, it is also to be appreciatedthat remote microphone devices configured to execute techniques fordescribed herein may have a number of other arrangements.

In certain examples, another device in the system directs the remotemicrophone device 103 (i.e., the adaptive processor 142) as to thedirection/area on which to focus. For instance, the sound processingunit 112 or a mobile device (e.g., application executed on a mobiledevice) could determine that direction, and communicate it to the remotemicrophone. Merely for ease of illustration, FIGS. 1A-1C illustrateexamples in which the operations of the remote microphone device 103(i.e., of the adaptive processor 142) to focus the microphone array 141on the recipient-specified region of interest/listening direction areenabled based on data received from the sound processing unit 112 (e.g.,from processing module 125). However, as noted, it is to be appreciatedthat these specific examples of FIGS. 1A-1C are merely illustrative.

More specifically, as noted above, the processing module 125 of thesound processing unit 112 includes the remote microphone focusing engine127. The remote microphone focusing engine 127 is configured todetermine a relative position (e.g., location and orientation) of theremote microphone device 103 to the cochlear implant 100 (e.g., thesound processing unit 112). In addition, the remote microphone focusingengine 127 is configured to determine the recipient-specified region ofinterest/listening direction based on one or more recipient providedindicia. As described further below, in certain embodiments therecipient provided indicia is a primary gaze (i.e., look) direction ofthe cochlear implant recipient.

Also as described further below, the remote microphone focusing engine127 is configured to cause the sound processing unit 112, morespecifically the wireless transceiver 111, to send “recipient-specifieddirectional data” to the remote microphone device 103, more specificallythe transceiver 145. The remote microphone device 103 (adaptiveprocessor 142) is configured to use the recipient-specified directionaldata to focus the microphone array 141 on the recipient-specified regionof interest/listening direction. The transfer of the recipient-specifieddirectional data from the sound processing unit 112 to the remotemicrophone device 103 is generally represented in FIG. 1A by arrow 143.

Although FIG. 1B illustrates the remote microphone focusing engine 127implemented at the sound processing unit 112, it is to be appreciatedthat this is one example arrangement. In alternative embodiments, theremote microphone focusing engine 127 may be implemented at a mobilecomputing device in communication with the sound processing unit 112 andthe remote microphone device 103 (e.g., as an application executed atthe mobile computing device). Alternatively, the remote microphonefocusing engine 127 may be implemented at the remote microphone device103. Additionally, it is to be appreciated that the remote microphonefocusing engine 127 could be implemented across a combination ofdevices, including two or more of the sound processing unit 112, theremote microphone device 103, a mobile computing device, or otherdevices operable with the sound processing unit 112 and the remotemicrophone device 103.

FIG. 2 is a high-level flowchart of a method 150 in accordance withcertain embodiments presented herein. For ease of description, method150 will be described with reference to cochlear implant system 101 ofFIGS. 1A-1C. In the embodiment of FIG. 2, the remote microphone device103 and the cochlear implant 100 are located within the same spatialarea. In addition, the cochlear implant 100 is worn by the recipient. Asused herein, the “same” spatial area refers to co-location of the remotemicrophone device 103 and the cochlear implant 100, and thus therecipient, within the same physical proximity. When the microphonedevice 103 and the cochlear implant 103 are within the same spatialarea, they are wireless connected with one another via a wirelesscommunications link, such as a short-range wireless connection (e.g., aBluetooth® link). Bluetooth® is a trademark of Bluetooth SpecialInterest Group (SIG), Inc.

Method 150 begins at 151 where a relative positioning (e.g., locationand orientation) of the remote microphone device 103 to the cochlearimplant 100 within the spatial area is determined. At 152, arecipient-specified region of interest with the spatial area and/or arecipient-specified listening direction is determined. At 153, using therelative positioning of the remote microphone device to the cochlearimplant 100, the direction of focus for the remote microphone device 103(i.e., how to focus the microphone array 141) is determined. At 154, theplurality of microphones 140 of the remote microphone device 103 (i.e.,the microphone array 141) are focused in the determined direction (i.e.,on the recipient-specified region of interest recipient-specifiedlistening direction). In other words, in the example of FIG. 2, theplurality of microphones 140 of the remote microphone device 103 areturned towards (e.g., have a directionality/sensitivity in a directionof) the recipient's preferred listening direction (i.e., therecipient-specified region of interest).

In FIG. 2, the operations at 154 are generally performed at the remotemicrophone device 103. However, the operations at 151, 152, and 153 ofFIG. 2 may be performed at the cochlear implant 100, the remotemicrophone device 103, a third device (e.g., mobile phone), or anycombination thereof. Further details regarding example operationsperformed at each of 151, 152, 153, and 154 are provided further.

Referring first to the operations at 152, a “relative position” of theremote microphone device 103 to the cochlear implant 100, or “relativepositioning” of the remote microphone device 103 to the cochlear implant100, may include several different elements/determinations, namelyrelative locations and orientations, and may be determined in a numberof different manners, several which are described below. Since, as notedabove, the sound processing unit 112 is an element of the cochlearimplant 100 (and thus is worn by the recipient), determining therelative positioning of the remote microphone device 103 to the soundprocessing unit 112 is to be understood to be the same as determiningthe relative positioning of the remote microphone device 103 to thecochlear implant 100.

In certain embodiments the remote microphone device 103 and the cochlearimplant 100 are able to determine their relative locations (e.g.,distance between the devices) and relative orientations (e.g., relativerotational angles) themselves. For instance, in certain embodiments, therelative positioning (e.g., location and orientation) of the remotemicrophone device 103 to the cochlear implant 100 may be based on one ormore compass readings, such as a compass reading from each of the soundprocessing unit 112 and the microphone device 103 (e.g., direction offocus relative to magnetic north or to a reference point in the room).In such embodiments that make use of compass readings, the remotemicrophone device 103 can focus in the same direction with a relativelywide angle.

In further embodiments, the relative positioning may includedetermination of the locations of each of the remote microphone device103 and cochlear implant 100 and/or determination of a distance betweenthe remote microphone device 103 and the cochlear implant 100. As noted,the relative positioning may also include determination of differencesin the orientations between the remote microphone device 103 and thecochlear implant 100 in one, two, or three dimensions/planes (e.g., inthe X, Y, and/or Z planes). It is to be appreciated that the locationsand orientations are “relative” in that they are not necessarilydetermined in terms of, for example, stationary coordinates. Instead, asdescribed further below, it is the relative separation between thedevices, and the angles at which the remote microphone device 103 andthe sound processing unit 112 (and thus the microphones thereof) areorientated, which may be used to dynamically adapt the operations of theremote microphone device 103. However, as described further below, incertain examples the relative locations and orientations of the remotemicrophone device 103 and the sound processing unit 112 may bedetermined with reference to a stationary reference point.

In accordance with certain embodiments, the relative positioning of theremote microphone device 103 and the cochlear implant 100 is a real-timeawareness of the relative locations/orientations of the two devices,which could change dynamically and/or regularly. As such, the relativepositioning of the remote microphone device 103 and the cochlear implant100 may be re-evaluated continually, periodically, etc. and, asdescribed below, may be determined and maintained through a referencepoint in the spatial area). As used herein, “real-time” should beinterpreted to include operations with or without some hysteresis,including certain attack times, release times, etc.

In certain embodiments, the relative locations of the remote microphonedevice 103 and the cochlear implant 100 could be determined, forexample: using a satellite positioning system (e.g., the United StatesNAVSTAR Global Positioning System (GPS)) in which satellites providegeolocation and time information to receivers in each of the devices;using wireless triangulation systems, sometimes referred to indoorpositioning systems (e.g., Wi-Fi® or Bluetooth® based systems), whichoperate by measuring the intensity/strength of signals received fromwireless access points, etc. Wi-Fi® is a registered trademark of theWi-Fi Alliance and Bluetooth® is a trademark of Bluetooth SpecialInterest Group (SIG), Inc.

In another example arrangement, one or more ranging pulses (e.g.,ultrasonic pulses, radio-frequency pulses, etc.) may be transmittedbetween the remote microphone device 103 and the sound processing unit112 to determine the relative location information (e.g., use pulses fordistance/range determination between the remote microphone device 103and the sound processing unit 112 worn by the recipient). Ultrasonicpulses, in particular, may take advantage of the plurality ofmicrophones present in many auditory prostheses and wireless devices.

In certain such examples, a sequence of ranging pulses are emitted fromthe remote microphone device 103. Range finding and direction detectiontechniques are then executed at the sound processing unit 112 todetermine the relative locations of the remote microphone device 103 andthe sound processing unit 112 (and thus the recipient wearing the unit).That is, certain embodiments may use ranging pulse emission from theremote microphone device 103 for analysis at the sound processing unit112. However, other embodiments may alternatively utilize ranging pulseemission from the sound processing unit 112 for analysis at the remotemicrophone device 103, an application executed at a third device (e.g.,mobile computing device), etc.

In certain arrangements, a single ranging pulse is transmitted relativeto a time reference and may be sufficient to determine the relativelocation information. In other embodiments, a sequence of closely spacedpulses could be transmitted and correlated with a known sequence ofpulses. This correlation between the received sequence and the knownsequence could be used to determine the distance between the twodevices, as well as the relative direction from the remote microphone103 to the sound processing unit 112 (e.g., the angles at which theremote microphone device 103 and the sound processing unit 112 areorientated). Additionally, certain embodiments may make use of theamplitude of the pulse(s), as received, to estimate direction. Incertain embodiments, a ranging pulse is transmitted from one device(e.g., the sound processing unit 112) to evoke a ranging pulsetransmission back from the other device (e.g., the remote microphonedevice 103), thereby enabling the relative location information to bedetermined.

In another example, a comparison of the sound environment at the soundprocessing unit 112 relative to the sound environment at the remotemicrophone device 103 could be used to determine the relative locationinformation. For example, as noted, each of the sound processing unit112 and the remote microphone device 103 are configured to capture soundsignals. These sound signals may be analysed for environmentaldifferences, which can be used to infer the relative locations of thesound processing unit 112 and the remote microphone device 103 withinthe spatial region. Certain embodiments may make use of a referencepoint for triangulation (either from acoustic mapping of the space oranother accessory).

As noted above, the sound processing unit 112 and the remote microphonedevice 103 may comprise orientation sensors 135 and 148, respectively.In certain embodiments, data generated by these sensors 135 and/or 148(e.g., relative movement of the devices from accelerometer data) may beanalyzed and used to determine the relative locations of the remotemicrophone device 103 and the sound processing unit 112. This analysiscould be performed at the sound processing unit 112 or another device,such as a mobile computing device (not shown in FIGS. 1A-1C) in wirelesscommunication with each of the sound processing unit 112 and the remotemicrophone device 103.

In still other embodiments, one or more image capture devices (cameras),such as cameras in a mobile computing device in communication with eachof the sound processing unit 112 and the remote microphone device 103,can be used to determine the relative location information. For example,stereo cameras could be used to capture image data which, when analyzed,could determine the distance between the sound processing unit 112 andthe remote microphone device 103. Additionally, a sequence of imagescould capture relative movement of the sound processing unit 112 and theremote microphone device 103. Such a sequence of images could beanalyzed using motion detection techniques to determine the relativedistance and direction information between the sound processing unit 112and the remote microphone device 103.

Referring next to determination of the relative orientation of theremote microphone device 103 to the sound processing unit 112, incertain examples the relative orientation information can be determinedalong with the location information, as described above (e.g., usingranging pulses, sensor data, image date, etc.). For example, in certainarrangements, the orientation information may be determined throughaccelerometer, gyroscope, compass magnetic sensors, etc. in each of theremote microphone device 103 and the cochlear implant 100 (e.g., soundprocessing unit 112). In certain embodiments, an accelerometer or othersensor could also be used to determine relative distance between thesound processing unit 112 and the remote microphone device 103. Forexample, referring specifically to an accelerometer, if the remotemicrophone device 102 was placed at a certain position, theaccelerometer could then be used to track the motion of the person asthey move away from that position. By summing together X, Y, Z motionalchanges, the relative distance information can be determined.

In certain examples, the relative positioning information (e.g.,relative location and orientation information) characterizes thedifferences in locations and orientations between the remote microphonedevice 103 and the cochlear implant 100 (e.g., sound processing unit112). However, in certain embodiments, a stationary reference point mayalso be set within the spatial area and the relative location andorientations between the remote microphone device 103 and thisstationary reference point, as well the relative location andorientations between the sound processing unit 112 and this stationaryreference point, can be determined. A stationary reference may be usedin embodiments in which the recipient and/or the sound source (e.g., aspeaker) move about the spatial area.

FIG. 3 is a schematic diagram illustrating the concept of a stationaryreference point 157 in a spatial area 155. As shown, the remotemicrophone device 103 and sound processing unit 112, worn by a recipient156 of the cochlear implant 100, are co-located in the spatial area 155.

As would be understood, the angle between the sound processing unit 112and a sound source 158 in the spatial area 155 will be different thanthe angle between the remote microphone device 103 and the sound source158. Additionally, it would be understood that these relative angles arebased on the locations and orientations of each of the sound processingunit 112 and the remote microphone device 103 at a given point in time.If the relative location and/or orientation of either device changes,then angles between the sound source 158 and each of the soundprocessing unit 112 and the remote microphone device 103 also change.The stationary reference point 157 and the relative locations andorientations of each of the remote microphone device 103 and the soundprocessing unit 112 thereto, can be used for orientation and/or locationsynchronization/calibration, where differential calculation is madebased on motion of the two devices relative to the reference point.

In certain examples, the reference point 157 can be, for example, afeature in the spatial area 155 (e.g., a corner, a piece of furniture,etc.). In other embodiments, the reference point 157 can be the locationof a reference device (e.g., a simple ultrasonic sender that emitultrasonic pulses and is placed some distance from the remote microphonedevice 103). With the reference point 157, and dynamic synchronizationchecks during use, the recipient 156 and/or the sound source 158 couldbe allowed to move around the spatial area 155. Additionally, the use ofthe stationary reference point 157 makes the techniques robust againstsomeone touching/moving the remote microphone device 103. However, it isto be appreciated that the use of a stationary reference point may notbe required in all embodiments, such as embodiments in which neither theremote microphone device 103 nor the recipient move about the spatialarea.

As noted above with reference to FIG. 2, after determining the relativepositioning (e.g., location and orientation) of the remote microphonedevice 103 to the cochlear implant 100, at 153 a recipient-specifiedregion of interest within the physical proximity is determined. Therecipient-specified region of interest may be determined/identified in anumber of different manners, some of which are described further below.

More specifically, in certain examples, the recipient-specified regionof interest may be determined based on a primary gaze (i.e., look)direction of the recipient. The primary gaze direction of the recipientmay be determined in a number of different manners. For example, in oneembodiment, one or more inertial measurements representing motion,including absence of motion, of the head of the recipient can becaptured using the one or more sensors 135. The inertial measurements(head motion data) can then be analyzed to determine the direction inwhich the recipient is primarily looking, which in turn is inferred tobe the region/direction of interest to the recipient (i.e., the portionof the spatial area from which sounds should be captured/emphasized).

Use of inertial measurements is one illustrative technique fordetermining a primary gaze direction of the recipient. In otherembodiments, the primary gaze direction of the recipient may bedetermined based on image data captured from external devices, such ascameras located in mobile computing devices (e.g., mobile phones). Incertain such examples, the image data, when analyzed using imageanalysis techniques, can be used to determine head and eye directions ofthe recipient. In certain examples, cameras could alternatively bebody-worn or placed around the room (e.g., a table, etc.) to captureimage data that be analyzed through image analysis techniques.

In certain examples, the image data may include faces and the imageanalysis techniques may automatically track individuals within thespatial region. That is, face recognition techniques can be used todetect the recipient and/or individuals who can be potential speakers(or actually are speaking). Moreover, in certain embodiments, if nopersons (potential sound sources) are detected at certain directions(certain spatial regions) from the remote microphone device 103, thenthe remote microphone device 103 can be instructed to ignore/removesound should come from those direction and remove sound receivedtherefrom (e.g., if no persons are in those regions, then the sounds maybe considered noise). The image data can, in certain embodiments, beused to determine who is speaking and thus enable the remote microphonedevice 103 to capture sound from that person even when the listener islooking in another direction.

In one specific example, an application may be executed (e.g., a mobilecomputing device) where, at start-up of the remote microphone device103, the recipient presses an activation button while at the samelooking in to the camera of the mobile computing device. The capturedimage data can be used to identify the recipient (i.e., the listener)for differentiation from other individuals who may be sources of soundto be captured by the remote microphone device 103.

Determining the recipient-specified region of interest based on aprimary gaze direction of the recipient is illustrative and othertechniques for determining the recipient-specified region of interestmay be used in other embodiments. In one such embodiment, therecipient-specified region of interest may be determined based on one ormore inputs received via a user interface of the one or more devices,where the user inputs indicate/identify at least one of a person or aregion on which the microphones 140 of the remote microphone device 103should be focused.

For example, FIG. 4 is a schematic diagram illustrating a touchscreen160 (i.e., user interface integrated with a display screen) of a mobilecomputing device 162 that may be used with the remote microphone device103 and cochlear implant 100, describe above. Although FIG. 4illustrates the use of a touch screen 162, other embodiments may usedifferent user interfaces (e.g., keyboard, touchwheel, etc.) that areseparate from a display screen.

Returning to the example of FIG. 4, the mobile computing device 162displays on the touchscreen 162 a visual representation 163 of thespatial area. This visual representation 163 identifies potential soundsources within the spatial area and, in certain examples, may alsoinclude a visual representation of the relative locations of the soundsources, the remote microphone device 103, and the recipient. In FIG. 4,the potential sound sources are different individuals 164 seated arounda table 165, wherein the remote microphone device 103 is positioned onthe top surface of the table 165.

In certain examples, the mobile computing device 162 performs an initialscene analysis of the spatial area, from the perspective recipient, togenerate the visual representation 163. This analysis may be based on,for example, sound data captured by the microphones 140 of the remotemicrophone device 104, sound data captured by sound input devices 108 ofthe sound processing unit 112, and/or other types of data captured byother input devices, such as image data captured by camera 166 of themobile computing device 162. Once the scene analysis is performed thevisual representation 163 is generated and displayed at the touchscreen162.

In certain examples, the visual representation 163 is an augmentedreality view of the spatial area. More specifically, using input devicesof the mobile computing device 162, such as the camera 166, orientationsensors, etc., the mobile computing device 162 generates and displays anenhanced live direct or indirect view of the real-world environment ofthe spatial region.

In the example of FIG. 4, the visual representation 163 of the spatialregion is augmented with a superimposed computer-generated message 168.In this illustrative example, the message 168 comprise instructionsstating “Tap Speaker to Focus,” which indicates to the recipient thatthe recipient should use the touchscreen 160 to identify the soundsource (i.e., individual 164) on which he/she would like the remotemicrophone device 103 to focus. Although not shown in FIG. 4, once therecipient selects an individual 164, that specific individual may behighlighted or otherwise identified within the visual representation163.

It is to be appreciated that the example of FIG. 4 illustrates asimplified user interface that enables a recipient to identify a personon which the recipient would like the remote microphone device 103 tofocus. It is to be appreciated the specific user interface of FIG. 4 isillustrative and that other user interfaces may be generated to enable arecipient to identify at least one of a person or a region within thespatial area as the recipient-specified region of interest. For example,in other embodiments, the user interface could include a representationof the remote microphone device 103 in the center, with different“listening zones” identified around the representation of the remotemicrophone device. In such an example, the recipient could tap on one ofthe “listening zones” to direct the remote microphone's beam to thatzone.

In other embodiments, the recipient-specified region of interest may bedetermined through microphone signal analysis, such as own voicedetection. For example, the direction in which the person speaks couldbe used to infer the recipient-specified region of interest (e.g., therecipient speaks towards region “A,” therefore the recipient wants tofocus on sounds coming from region “A”).

It is to be appreciated that the different techniques for determiningthe recipient-specified region of interest described above are notmutually exclusive and that these different techniques may be usedtogether in different combinations. For example, in one sucharrangement, the recipient-specified region of interest could initiallybe determined based on inputs received via a user interface. Therecipient-specified region of interest could then be dynamically updatedbased on the subsequent primary gaze direction of the recipient.

In addition, certain embodiments presented herein may use signalprocessing to provide an initiation synchronization between the soundprocessing unit 112 and the remote microphone device 103. For example,the remote microphone device 103 may capture “snapshots” (e.g., capturedaudio signals captured within a discrete time window) of the auditoryscene in a number of directions around the entirety of the device. Thesesnapshots obtained at the remote microphone device 103 may be comparedto the auditory scene from the perspective of the recipient (e.g., asdetermined by the sound processing unit 112 also using snapshots of theauditory scene in a number of directions). Once the best match is found,the devices are synchronized. Synchronization may be maintained throughrepeated use of snapshots.

As noted above with reference to FIG. 2, after determining therecipient-specified region of interest, the relative location andorientation of the remote microphone device 103 and the cochlear implant100 is used to determine a direction of focus of the microphones of theremote microphone device and, eventually, the plurality of microphones140 of the remote microphone device 103 are focused in that direction(i.e., on the recipient-specified region of interest/listeningdirection). More specifically, using the relative location andorientation of the remote microphone device 103 and the cochlear implant100, the cochlear implant and/or the mobile computing determines adesired direction (or a location of an audio source), relative to theremote microphone device 103, to which the plurality of microphones 140should be focused (e.g., have increased sensitivity to). The dataindicating the recipient-specified direction, or recipient-specifiedarea of interest, is sometimes referred to herein as“recipient-specified directional data,” which is sent from the cochlearimplant and/or the mobile computing to the remote microphone device 103.Using the recipient-specified directional data, the remote microphonedevice 103 adapts its operations (e.g., at adaptive processor 142) sothat the sensitivity of the microphones 140 (i.e., the microphone array141) is towards the recipient-specified region of interest.

In accordance with certain embodiments presented herein, thedetermination of the recipient-specified region of interest/listeningdirection, as opposed to any default, may rely upon some attack/releasetimes suited to typical listening habits, the speed of head movements,or other factors. For example, in some embodiments, the direction offocus of the remote device 103 should not change if the recipientquickly glances at a clock or other direction for only a brief period oftime and/or if recipient quickly glances in a direction without a soundsource. In other words, quick head movements and/or brief re-directionsof focus need not trigger a direction/focus update on the remotemicrophone device 103. Similarly, the attack/release times may beselected so that the techniques are robust against the recipient leaningback/forth/to the sides, where such movement can change the distance andangle between the sound processing unit 112 and the remote microphonedevice 103, as well as the distance and angle between the soundprocessing unit 112 and the sound source.

Additionally, the cochlear implant 100 (or mobile device) and/or remotemicrophone device 103 can determine whether a direction of focusincludes a legitimate sound source (e.g., speech, good SNR, etc.),noise, or no sounds before focusing the microphones 140 in thatdirection. If it is determined that there is only noise or no sounds inthe recipient-specified region of interest, the cochlear implant 100 notupdate the remote microphone device 103 (i.e., will not send therecipient-specified directional data) or the remote microphone device103 will note execute a direction/focus update (i.e., will not act onthe recipient-specified directional data). Similarly, the cochlearimplant 100 and/or the remote microphone device 103 can scan the roominitially and/or periodically to identify where troubling noise sourcesare located and filter those out expressly even as the recipient'sdirection of focus changes.

In certain embodiments, the determination of the recipient-specifiedregion of interest could also be applied in a negative/subtractivemanner, meaning that this information could be used to cancel certainspeakers or sound sources in the space. Thus, sound from any directionin which the recipient's focus does not linger could be treated asnoise.

The adaptive microphone focusing at the remote microphone device 103 maybe implemented in a number of different manners. For example, in someinstances, recipients may benefit from a remote microphone device withmicrophones in an omnidirectional mode. So, in certain embodimentspresented herein, the remote microphone device has different modes ofoperation that are selectable automatically based on therecipient-specified data, or which may be autonomouslyselected/configured based on an assessment of the listening situationby, for example, the sound processing unit 112, a mobile computingdevice, etc.

In certain examples, information from an environmental classifier on thecochlear implant 100 and/or the remote microphone device 103 can be usedas a factor in the adaptions at the remote microphone device 103 (e.g.,the adaptive processor 142). For example, different listeningenvironments (e.g., speech in noise, noise, quiet, speech, etc.) canbenefit from different adaptions. For example, difficult listeningenvironments (e.g., speech in noise, noise, etc.) might not require moreaggressive adaptions (e.g., noise reductions) than easy listeningenvironments (e.g., quiet, speech, etc.). In such difficultenvironments, the recipient might prefer to hear all speakers around atable with the microphones 140 on the remote microphone device 103 inomnidirectional mode, but also be able to dynamically switch in and outof the focusing at certain times. In addition, poor SNR in the directionof focus (i.e., towards the recipient-specified region of interest)could lead to enabling additional processing operations and/or disablingthe recipient-specified focusing of the microphones.

In certain embodiments, the remote microphone device 103 can operate inomnidirectional mode and continue to analyze the recipient's directionof focus and make adjustments as needed. Further, very frequent headmovements or frequent and significant head movements could benefit fromomnidirectional mode. Thus, accelerometers in the cochlear implant 100could drive enablement/disablement of the adaptive features.

FIG. 5 is flowchart of one example method 170 in accordance with certainembodiments presented herein. Method 170 begins at 172 where an auditoryprosthesis is paired with a remote microphone device that comprises aplurality of microphones forming a microphone array. That is, theauditory prosthesis and the remote microphone device are registered withone another so that the two devices can wirelessly communicate with oneanother when they are within a relative physical proximity to oneanother.

At 173, the remote microphone device is placed at some location within aspatial area that is remote to (i.e., some distance from) the recipientand, at 174, the relative positioning (e.g., location and orientation)of the auditory prosthesis and the remote microphone device isdetermined. At 175, the remote microphone device is activated and beginsto capture sounds. Initially, the remote microphone device may capturesounds using some default (e.g., omnidirectional) settings.

At 176, a determination is made as to whether or not the remotemicrophone device should focus its microphone array on arecipient-specified direction. This determination may be based, forexample, in part on whether or not recipient-specified direction datahas been provided by the auditory prosthesis or another device (e.g.,mobile computing device paired with the auditory prosthesis). As notedabove, this determination may also be based on other environmental orsound information, information from other sensors, etc.

If it is determined that the remote microphone device should not focusthe microphone array on a recipient-specified direction/area, then at177 the remote microphone device continues to capture sounds using thedefault settings. At 178, a determination can be made as to whether ornot it is time to re-evaluate the relative positioning (e.g., locationand orientation) of the auditory prosthesis and the remote microphonedevice. If it is time to re-evaluate the relative positioning of theauditory prosthesis and the remote microphone device, then method 170returns to 174. Otherwise, method 170 returns to 176. It is to beappreciated that the operations at 178 may be performed, for example,periodically and in certain examples method 170 proceeds from 177directly back to 176.

Returning to 176, if it is determined that the remote microphone deviceshould focus the microphone array on a recipient-specifieddirection/area, then at 179 the operations of the remote microphonedevice are adapted so that the microphone array focuses in (i.e., hasincreased sensitivity to) the recipient-specified direction/area. At180, the remote microphone device captures sounds while focusing on therecipient-specified direction/area. Method 170 may then again proceed to178 for a determination as to whether or not it is time to re-evaluatethe relative positioning of the auditory prosthesis and the remotemicrophone device. As noted, method 170 may then return to 174 or 176,depending on the results of the determination. If it is time tore-evaluate the relative positioning of the auditory prosthesis and theremote microphone device, then method 170 returns to 174. It is to beappreciated that the operations at 178 may be performed, for example,periodically and in certain examples method 170 proceeds from 180directly back to 176.

FIG. 5, as well as other embodiments presented herein, have primarilybeen described with reference to focusing of the microphone array of aremote microphone device on a recipient-specified direction/area.However, in certain embodiments, the best microphone(s) available couldactually be the ones on the auditory prosthesis, rather than those onthe remote microphone device (e.g., in case the discussion partner ispositioned right next to the recipient). As such, an auditory prosthesissystem in accordance with certain embodiments presented herein may focusthe microphones (microphone array) present on the auditory prosthesis onthe recipient-specified direction/area. This focusing may be instead of,or in addition to, the focusing of the microphone array of the remotemicrophone device.

For example, in one such embodiment, an auditory prosthesis systemcomprises an auditory prosthesis configured to be worn by a recipient, aremote microphone device and, in certain cases, a mobile computingdevice, all located in the same spatial area. The auditory prosthesisincludes a first microphone array, while the remote microphone deviceincludes a second microphone array. In these examples, one or moreprocessors are configured to synchronize location and orientationinformation of the remote microphone device with location andorientation information of the auditory prosthesis. The one or moreprocessors are also configured to determine, based on the synchronizedlocation and orientation information, a recipient-specified region ofinterest within the spatial area and to focus one or both of the firstmicrophone array or the second microphone array on therecipient-specified region of interest within the spatial area. The oneor more processors may be provided on/in the auditory prosthesis, theremote microphone device, the mobile computing device, or anycombination thereof.

As described above, certain embodiments presented herein may make use ofimage data captured from external devices, such as cameras located inmobile computing devices to determine a recipient-specified direction orarea of interest (e.g., as indicated by a primary gaze direction of arecipient). However, in certain embodiments presented herein, a remotemicrophone device may include a camera that is configured to captureimage data. In such embodiments, the remote microphone device may alsoinclude one or more processors configured to analyze image data todetermine the recipient-specified direction/area of interest (e.g., asindicated by a primary gaze direction of a recipient). In suchembodiments, since the remote microphone device itself is configured todetermine the recipient-specified direction/area of interest, no orlimited interaction with a sound processing unit (beyond sendingcaptured sound data to thereto) is needed (e.g., no need for gyroscope,accelerometer, focusing engine, etc. in the associated sound processingunit). Further details of such an embodiment are described below withreference to FIGS. 6A and 6B.

More specifically referring first to FIG. 6A, shown is a remotemicrophone device 603 positioned proximate to a recipient 656. Theremote microphone device 603 comprises a microphone array 641 formed bythree microphones 640, a camera 682, and a pairing button 684, amongother elements. In this example, the camera 682 is a 360° fisheye cameraand the remote microphone device 603 is configured to perform facerecognition and to implement a person tracking system that maintains arecord and awareness of all persons in a spatial area. In addition, theremote microphone device 603 is configured to dynamically add or removepersons from the tracking system.

Before using the remote microphone device 603, is paired with a soundprocessing unit for audio streaming of captured sound data thereto. Thisprocess is sometimes referred to herein as “device pairing.” Inaddition, the remote microphone device 603 is also “face-paired” with arecipient or recipients. As used herein, “face-pairing” refers to aprocess in which the remote microphone device 603 learns/identifies therecipient with which the remote microphone device 603 is associated(i.e., which individual/person for which the remote microphone device603 should determine a recipient-specified direction/area of interest).In practice, the device pairing may be performed in advance or at thesame time as the face pairing may each may be performed using thepairing button 684 via, for example, different button press sequences.In an alternative embodiment, different pairing buttons may be providedfor each of the device pairing and the face-pairing processes.

FIG. 6B illustrates operation of the remote microphone device 603 tocapture sound signals. More specifically, during operation, the remotemicrophone device 603 analyzes image data captured by the camera 682 andfollows the recipient's face, and potentially the faces of other personsin the spatial area. The remote microphone device 603 calculate thedistance to the recipient and the angle where he/she is looking (i.e.,the gaze direction). The recipient's gaze direction, shown as 686, isthen used to control the beam forming for the microphone. In FIG. 6B,the resulting directionality (focus) of the microphone array 641 is amicrophone beam 688.

In certain examples, the remote microphone device 603 may be face-pairedwith multiple recipients simultaneously. In such examples with multiplerecipients, the result is multiple simultaneous microphone beams andmultiple simultaneous audio streams to different sound processing units.In certain examples, the data from camera 682 also be used to identifysound sources that should be cancelled/ignored (e.g., sound coming fromdirections with no faces, directions of traffic noise from a window orsimilar, etc.). The data from camera 682 may also be used to recognizefaces provided, for example, on a video/display screen.

It is to be appreciated that the embodiments presented herein are notmutually exclusive.

The invention described and claimed herein is not to be limited in scopeby the specific preferred embodiments herein disclosed, since theseembodiments are intended as illustrations, and not limitations, ofseveral aspects of the invention. Any equivalent embodiments areintended to be within the scope of this invention. Indeed, variousmodifications of the invention in addition to those shown and describedherein will become apparent to those skilled in the art from theforegoing description. Such modifications are also intended to fallwithin the scope of the appended claims.

What is claimed is:
 1. A method, comprising: determining a relativelocation and a relative orientation of a remote microphone device to anauditory prosthesis, wherein the auditory prosthesis and the remotemicrophone device are each located within a same spatial area andwherein the remote microphone device comprises a plurality ofmicrophones; determining a recipient-specified region of interest withinthe spatial area; and using the relative location and the relativeorientation of the remote microphone device to the auditory prosthesisto focus the plurality of microphones of the remote microphone device onthe recipient-specified region of interest within the spatial area. 2.The method of claim 1, wherein determining a recipient-specified regionof interest within the spatial area comprises: determining a primarygaze direction of a recipient of the auditory prosthesis; andcorrelating the primary gaze direction of the recipient with the spatialarea to determine the recipient-specified region of interest.
 3. Themethod of claim 2, further comprising: capturing one or more inertialmeasurements representing motion of the head of the recipient of theauditory prosthesis; and determining the primary gaze direction of therecipient based on the one or more inertial measurements.
 4. The methodof claim 2, further comprising: capturing, via one or more image capturedevices, image data from the spatial area; and performing image analysistechniques to determine the primary gaze direction of the recipientbased on the image data.
 5. The method of claim 1, wherein determining arecipient-specified region of interest within the spatial areacomprises: receiving, via a user interface, one or more inputsidentifying at least one of a person or a region of the spatial area. 6.The method of claim 1, further comprising: determining the relativelocation of the remote microphone device to the auditory prosthesis inrelation to a stationary reference point in the spatial area.
 7. Themethod of claim 1, further comprising: capturing, using the plurality ofmicrophones of the remote microphone device, sound signals from therecipient-specified region of interest; and wirelessly sending the soundsignals to the auditory prosthesis.
 8. The method of claim 1, whereinthe relative location of the remote microphone device to an auditoryprosthesis and the recipient-specified region of interest within thespatial area are determined at the auditory prosthesis, and wherein themethod further comprises: sending recipient-specified directional dataindicating the recipient-specified area of interest from the auditoryprosthesis to the remote microphone device; and using therecipient-specified directional data at the remote microphone device tofocus the plurality of microphones of the remote microphone device onthe recipient-specified region of interest.
 9. The method of claim 1,wherein the relative location of the remote microphone device to anauditory prosthesis and the recipient-specified region of interestwithin the spatial area are determined at a mobile computing device inwireless communication with the auditory prosthesis and the remotemicrophone device, and wherein the method further comprises: sendingrecipient-specified directional data indicating the recipient-specifiedarea of interest from the mobile computing device to the remotemicrophone device; and using the recipient-specified directional data atthe remote microphone device to focus the plurality of microphones ofthe remote microphone device on the recipient-specified region ofinterest.
 10. A method, comprising: synchronizing location andorientation information of a remote microphone device with location andorientation information of an auditory prosthesis worn by a recipient,wherein the auditory prosthesis and the remote microphone device areeach located within a same spatial area; determining, based on thesynchronized location and orientation information, a physical separationbetween the remote microphone device and the auditory prosthesis andorientation angles of each of the remote microphone device and theauditory prosthesis; determining a substantially real-timerecipient-specified listening direction for the remote microphonedevice; and wirelessly sending directional data to the remote microphonedevice indicting the substantially real-time recipient-specifiedlistening direction, the physical separation between the remotemicrophone device and the auditory prosthesis, and the orientationangles of each of the remote microphone device and the auditoryprosthesis.
 11. The method of claim 10, further comprising: based on thedirectional data, configuring a microphone array of the remotemicrophone device for increased sensitivity to sounds originating fromthe recipient-specified listening direction.
 12. The method of claim 10,wherein determining a recipient-specified listening direction comprises:determining a primary gaze direction of the recipient; and correlatingthe primary gaze direction of the recipient with the location andorientation information of a remote microphone device.
 13. The method ofclaim 12, further comprising: capturing one or more inertialmeasurements representing motion of the head of the recipient of theauditory prosthesis; and determining the primary gaze direction of therecipient based on the one or more inertial measurements.
 14. The methodof claim 12, further comprising: capturing, via one or more imagecapture devices, image data from the spatial area; and performing imageanalysis techniques to determine the primary gaze direction of therecipient based on the image data.
 15. The method of claim 10, whereindetermining a recipient-specified listening direction comprises:receiving, via a user interface, one or more inputs identifying at leastone of a person or a region of the spatial area.
 16. The method of claim10, further comprising: synchronizing the location and orientationinformation of the remote microphone device with the location andorientation information of the auditory prosthesis in relation to astationary reference point in the spatial area.
 17. The method of claim11, further comprising: capturing, using the increased sensitivity ofthe microphone array, sounds originating from the recipient-specifiedlistening direction; and wirelessly sending the sound signals to theauditory prosthesis.
 18. The method of claim 10, further comprising:determining the physical separation between the remote microphone deviceand the auditory prosthesis and the orientation angles of each of theremote microphone device and the auditory prosthesis at the auditoryprosthesis.
 19. The method of claim 10, further comprising: determiningthe physical separation between the remote microphone device and theauditory prosthesis and the orientation angles of each of the remotemicrophone device and the auditory prosthesis at a mobile computingdevice.
 20. An auditory prosthesis system, comprising: an auditoryprosthesis; a remote microphone device; and one or more processorsconfigured to determine a relative location difference between theremote microphone device and the auditory prosthesis, determine arelative angular orientation difference between the remote microphonedevice and the auditory prosthesis, and to determine arecipient-specified listening direction, wherein the remote microphonedevice comprises: a wireless transceiver configured for direct orindirect communication with an auditory prosthesis; a microphone arrayconfigured to capture sound signals; and at least one processorconfigured to process the captured sound signals for wirelesstransmission by the wireless transceiver to the wireless transceiver ofthe auditory prosthesis, wherein the at least one processor isconfigured to use the relative location difference between the remotemicrophone device and the auditory prosthesis and the relative angularorientation difference between the remote microphone device and theauditory prosthesis to dynamically adjust a directionality of themicrophone array to the recipient-specified listening direction for theremote microphone device.
 21. The auditory prosthesis system of claim20, wherein the relative location difference between the remotemicrophone device and the auditory prosthesis and the relative angularorientation difference between the remote microphone device and theauditory prosthesis is identified in directional data received at theremote microphone device from at least one of the auditory prosthesis ora mobile computing device, wherein the directional data furtherindicates the recipient-specified listening direction for the remotemicrophone device.
 22. The auditory prosthesis system of claim 21,wherein to adjust the directionality of the microphone array based ondirectional data, the at least one processor is configured to performbeamforming on the sound signals captured by the microphone array toincrease the sensitivity of the microphone array towards the at leastone of the recipient-specified area of interest or therecipient-specified listening direction.
 23. The auditory prosthesissystem of claim 21, wherein the at least one processor is configured touse the directional data to cancel sound signals received at themicrophone array from one or more directions.
 24. The auditoryprosthesis system of claim 20, wherein the at least one processor isconfigured to perform a classification of the audio environment of theremote microphone device, and wherein the at least one processor isconfigured to dynamically adjust a directionality of the microphonearray further based on the classification of the audio environment. 25.The auditory prosthesis system of claim 20, wherein the at least oneprocessor is configured to determine whether the one of arecipient-specified area of interest within a spatial area of the remotemicrophone device or a recipient-specified listening direction for theremote microphone device corresponds to a location of a valid soundsource in the spatial area, and wherein the at least one processor isconfigured to dynamically adjust a directionality of the microphonearray only when the one of the recipient-specified area of interestwithin a spatial area of the remote microphone device or therecipient-specified listening direction for the remote microphone devicecorresponds to a location of a valid sound source.
 26. The auditoryprosthesis system of claim 20, further comprising a camera, wherein theat least one processor is configured to determine the one of arecipient-specified area of interest within a spatial area of the remotemicrophone device or a recipient-specified listening direction for theremote microphone device based on image data captured via the camera.27. The auditory prosthesis system of claim 26, wherein the at least oneprocessor is configured to determine a gaze direction of a recipient ofthe auditory prosthesis.
 28. The auditory prosthesis system of claim 21,wherein the auditory prosthesis is configured to wirelessly send thedirectional data to the remote microphone device.
 29. The auditoryprosthesis system of claim 21, wherein the one or more processors arepart of a mobile computing device, wherein the mobile computing deviceis configured to wirelessly send the directional data to the remotemicrophone device.